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Abstract — In Reliable Communication in the Absence of a 
Common Clock (Yeung et al., 2009), the authors introduce general 
run-length sets, which form a class of constrained systems that 
permit run-lengths from a countably infinite set. For a particular 
definition of probabilistic capacity, they show that probabilistic 
capacity is equal to combinatorial capacity. In the present work, 
it is shown that the same result also holds for Shannon's original 
definition of probabilistic capacity. The derivation presented 
here is based on generating functions of constrained systems as 
developed in On the Capacity of Constrained Systems (Bocherer et 
al., 2010) and provides a unified information-theoretic treatment 
of general run-length sets. 

I. Introduction 

A constrained system allows the noiseless transmission 
of input sequences of weighted symbols that fulfill certain 
constraints on the symbol constellations. The weight of a 
symbol can have different practical meanings, e.g., in the 
context of magnetic recording systems, "weight" usually refers 
to "tape-length"; other meanings like "time" or "energy" are 
possible, depending on the modeled system. 

For the design of encoders for such systems, it is of 
interest to determine the maximum entropy rate at which a 
random process can generate strings that fulfill the constraints. 
This rate is called the probabilistic capacity and a process 
that reaches this rate is called maxentropic . The probabilistic 
capacity is equal to the combinatorial capacity in the case that 
the constraints form a regular language. This was originally 
shown in (TJ. In 0, the authors show this property for a 
slightly generalized setup, since they allow non-integer valued 
symbol weights, as long as the set of weights is not too dense. 
The precise definition of "not too dense" is stated in SectionlTil 

Recently, in the authors introduced the continuous 
time asynchronous channel as a model for time jitter in a 
communication system with no common clock between the 
transmitter and the receiver. As a constrained system, such 
a channel is defined by an in general countably infinite set 
of distinguishable run-lengths W and a finite set of labels 
C. A run is a substring during which the label does not 
change. The channel then allows the noiseless transmission 
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of strings where each run has a length in W and a label in 
C and where in addition two subsequent runs always have 
different labels. We will refer to such channels by general 
run-length sets. Formally, general run-length sets are different 
from constrained systems that have been considered so far 
in two ways: first, the set of strings that are accepted do in 
general not form a regular language, and second, the set of 
distinct string-lengths does in general not fulfill the "not too 
dense" restriction. As a result, general run-length sets cannot 
be treated using the framework of finite state machines 0. 

In 0, we presented a new framework for constrained 
systems based on generating functions. Differing from 0, 
this framework is neither restricted to regular constraints nor 
does it impose the "not too dense" restriction to the set 
of string-lengths. The key result of [4] is that the entropy 
rate of input processes is in general upper-bounded by the 
combinatorial capacity of the system. This result allows us to 
derive the equality of probabilistic and combinatorial capacity 
for general run-length sets in a unified manner: we first derive 
the generating function of general run-length sets, we then 
use the generating function to calculate the combinatorial 
capacity, and we finally define an input process whose entropy 
rate is equal to the combinatorial capacity and thus achieves 
the maximum. In exactly the same way we derived in 
as an illustrating example the equality of probabilistic and 
combinatorial capacity for (j, k) constraints. 

For a particular definition of probabilistic capacity, the 
equality of probabilistic and combinatorial capacity of general 
run-length sets was derived in yj. The definition of probabilis- 
tic capacity in [3] is based on a notion of entropy rate that is 
different from Shannon's in Appendix 4]. Because of the 
different definitions of probabilistic capacity, the derivations 
presented in our work strongly differ from the derivations in 
0. We discuss the relation of our results to the results in [3] 
in Subsection IVI-DI 

The remainder of this work is organized as follows. In 
Section |nj we state the results from [4] that we need in this 
work. In Section [Till we formally define general run-length 
sets. In Section lTVllVl and lVIl we derive for general run-length 
sets the generating function, the combinatorial capacity, and 
the probabilistic capacity, respectively. 



II. Constrained Systems, Generating Functions, 
and Capacity 

In this section, we shortly provide the main definitions 
and results from our work (4). These form a mathematical 
framework based on generating functions for the information 
theoretic treatment of constrained systems. Within this frame- 
work, we will derive the probabilistic capacity of general run- 
length sets in the remaining sections. A detailed discussion 
and proofs of the theorems can be found in |4). 

A. Constrained Systems and Generating Functions 

Definition 1. A constrained system (A, w) consists of a 
countable set A of strings accepted by the system and an 
associated weight function w: A — > R>o (R>o denotes the 
positive real numbers) with the following property: if a, b G A 

and cat(a, b) G A then w[cat(a, b)] = u;(a) + w(b). 

Here and hereafter, cat (a, b) denotes the concatenation of 
a and b. 

Definition 2. Let (A, w) represent a constrained system. We 
define the generating function of A by 

G A (s) := J2 e ~ W (Q)S > S ^ C (D 

where C denotes the set of complex numbers. 

Let O = {w(a)|a 6 .4} denote the set of distinct string 
weights of elements in A. We order and index the set il such 
that £1 = with V\ < 1/2 < ■ ■ ■ . For every Vi 6 O, 

N{vi) denotes the number of distinct strings of weight in 

A. A compact representation of the sequence {N {vi)}°^ 1 can 
be obtained by rewriting the generating function as 

00 

G A (s) = Y J N(v l )e-»> s . (2) 

i=l 

The growth of the sequence {N (vi)}°^ 1 is determined by the 
analytic properties of G A (5). This property lies at the heart 
of the results presented in this section. 

B. Combinatorial Capacity 

From an information-theoretic perspective, of main interest 
is the exponential growth of the sequence {N which 
is captured by the notion of "combinatorial capacity". 

Definition 3. We define the combinatorial capacity by 

C := limsup — — (3) 

fc— >oo v k 

Here and hereafter, In denotes the natural logarithm. Shan- 
non's original definition of combinatorial capacity was 

C : = lim sup 1 v n . (4) 

The original definition is only meaningful when the set of 
distinct string weights il is not too dense, that is, when there 



exists some constant L > and some constant K > such 
that for any integer n > 

max k < Ln K . (5) 

v k <n 

See and (6) for detailed discussions of the "not too dense" 
property. The following theorem shows how the combinatorial 
capacity of a constrained system is related to its generating 
function and it further shows that our definition of combina- 
torial capacity is consistent with the original one. 

Theorem 1. Let {A, w) be a constrained system with the 
generating function G A {s). The following holds: 

1) The combinatorial capacity C is equal to the abscissa 
of convergence Q of G A , is., C = Q. 

2) If the set of distinct string weights Q is not too dense, 
then Co = Q and in particular Co = C. 

C. Entropy Rate of Input Processes 

In consistency with [1 Appendix 4], we define the entropy 
rate of weighted random processes. 

Definition 4. The entropy rate of a random process 
Y = {Yi}°^ 1 , Yi G y with an associated weight function 
w: y — > M>o is defined as 

H(Y) := limsup H(Y U . . . ,Y k ) 
1 ' k^oo E[w(Yi) -\ h w(Y k )} 

The operators E, H are defined as follows: For a discrete 
random variable X and a deterministic function /, E[/(X)] 
denotes the expected value of f(X) with respect to the prob- 
ability mass function (PMF) of X, and R(X) := E[- ln(X)] 
denotes the entropy of X. 

Let Y = {Yi} c *L 1 denote a random process that generates 
input for a constrained system. Transmitted over the system 
are the strings cat(Yi, . . . ,lfe). To ensure that the entropy rate 
as defined in Definition [4] actually reflects the entropy rate of 
the transmitted strings, we have to ensure that Y generates 
the strings unambiguously, see |4] Section V]. The following 
definition guarantees this. 

Definition 5. Let Y = {ii}g x , Yi G y be a random 
process and let pk denote the PMF of (Yi, . . . , YJ-). Define the 
sequence of auxiliary random variables Xk = cat(Yi, . . . , Yk) 
with the supports truncated to 

Af fc = {cat(j/i, . . .,y k )\(yi, . . .,y k )ey k : p k (yi, ■ ■ .,y k )>0} 

where y k denotes the Cartesian product of k copies of y. The 
process Y is an input process of the constrained system (A, w) 
if the supports Xu fulfill both of the following conditions: 

2) if j ■£ k, then Xj D X k = 0. 

Condition 1) ensures that Y generates valid strings and 
condition 2) ensures that Y does so unambiguously. For a 
discussion of these conditions in the context of general run- 
length sets, see Subsection IVI-AI in this work. The entropy 
rate of an input process of a constrained system relates to the 
combinatorial capacity as follows. 



Theorem 2. Let {A, w) denote a constrained system. The 
entropy rate of an input process Y of {A, w) is upper-bounded 
by the abscissa of convergence Q ofG^, and, in particular, it 
is upper bounded by the combinatorial capacity C of (A, w). 

Having defined entropy rate in Definition [4] and input pro- 
cesses in Definition [5] we implicitly defined the probabilistic 
capacity of a constrained system: it is given by the maximum 
entropy rate an input process can have. 

III. Setup 

A. Definition of (W, C) 

The class of constrained systems that we consider in this 
work can be specified by a set W of run-lengths and a set L 
of labels. The set of run-lengths W is a non-empty, countable 
subset of the positive real numbers R>o- The set of labels L 
is nonempty and finite. One run is the substring of a string 
during which the label does not change. We refer to the length 
of a run r by its weight w(r) and we refer to the label of a 
run r by the label function l(r). The set of allowed strings of 
such a system is given by 

(W,£) :={cat(n,...,r fc ) | k e N, 

for i = 1, . . . , k: w{ri) G W and l(n) G C, 
fori = l,...,k-l: l(r i+1 ) ? l(n)} (7) 

where N = {1,2,...} denotes the natural numbers. From this 
definition, we see that if the cardinality of C is equal to one, 
then each string in (W, C) consists of only one run with its 
length in W and with its label equal to the unique label from 
C. From now on, we therefore assume \C\ > 2. We do not 
require that the set (W, C) fulfills the "not too dense" property 
©. However, we assume that the limit in (01 exists, i.e., that 
(W, C) has a well-defined combinatorial capacity. 

B. (W, C) is in General not Regular 

It is important to note that (W, C), in general, does not form 
a regular language. If it would, it could be analyzed within the 
framework of finite state machines as defined in [2]. Consider 
as an example (V, JC) where 

V = {2 fc |fcGN} (8) 
/C={0,1}. (9) 

This is for A = N and £ = 2 an example for the asynchronous 
[A,£] channel with binary input as introduced in [3|. 

The first way to argue that (V, JC) is not regular is to 
interpret the set of runs 

{r\w{r) G V, l{r) G JC} (10) 

as an infinite alphabet. The set (V, JC) is thus generated by an 
infinite alphabet, whereas a regular language is by definition 
generated by a finite alphabet Q. However, the set (V,JC) 
can also be generated by concatenating the two runs ro,r\ 
with w(ro) = w(ri) = 1 and l(ro) = 0, l(ri) = 1. Indeed 
we now have a finite alphabet {ro,ri} by which we can 
generate (V, fC), but we need an infinite memory: only the 



concatenation of 1, 2, 4, 8, . . . runs of the same label result in 
a valid run-length. This memory cannot be implemented by 
a finite-state machine, which again shows that (V, JC) is not 
regular. 

C. How to Derive tlie Probabilistic Capacity of (W, C) 

Our aim is to derive the probabilistic capacity of (W, C) 
in the general case, that is, we want to derive the maximum 
entropy rate an input process of (W, C) can have. According to 
Theorem[2] the maximum entropy rate is upper bounded by the 
combinatorial capacity, which itself is, by Theorem [T] given 
by the abscissa of convergence of the generating function. Our 
approach is now as follows: 

1) We first derive the generating function. 

2) We then use the generating function to calculate the 
combinatorial capacity. 

3) We finally define an input process whose entropy rate is 
equal to the combinatorial capacity. 

After accomplishing these three tasks, we have shown that 
the probabilistic capacity of (W, C) is equal to its combi- 
natorial capacity and that its value can be derived by using 
the corresponding formulas for the combinatorial capacity. It 
should be noted that this approach is also suitable for other 
types of constrained systems. For instance, it was used in [4] 
to derive the combinatorial and the probabilistic capacity of 
(J, k) constraints. 

IV. Generating Function of (W, C) 

We start by deriving the generating function of (W, C). We 
do this run by run. The generating function of the run-length 
of the first run is given by 

For the first run, we can choose from among \L\ labels, so the 
generating function of the first run R\ is given by 

G fil (s) = |£|G w ( S ). (12) 

The generating function of the run-lengths of the second run is 
again Gyy (s), however, given the label chosen for the first run, 
we can for the second run only choose from among \L\ — 1 
labels. The first two runs result from concatenating the first 
run with the second run. Since we guarantee that two subse- 
quent runs have different labels, concatenating the first with 
the second run corresponds to multiplying the corresponding 
generating functions J8] Chapter 2]. The generating function 
of the first two runs is thus 

G R M = \C\ G w ( s )(|£| - 1) G w (s). (13) 

The same as for the second run applies for all subsequent runs, 
so the generating function of the first k runs is given by 

G Rk (s) = |£|G w (s)[(|£| - l)Gw(s)]^ 1 (14) 



To get the complete generating function of (W, £) we have 
to add up all Gu k . 

oo 
k=l 

oo 

= \C\G w ( s )Y / [^\-^)Gw(s)] m . (16) 

V. Combinatorial Capacity of (W, £) 

With the help of the generating function ( TTol l as derived 
in the previous section, we can now derive the combinatorial 
capacity of (W, £) by applying TheoremQ] Let Re(s) denote 
the real part of s. From [9 Theorem 3], we know that 
G{w,£}( s ) converges if and only if G(w.£) P- e ( s )] converges, 
and since G(w £) is strictly positive on the real axis, the latter 
converges if and only if 

(|£| - l)G w [Re(s)] < 1. (17) 

Thus, as a corollary to Theorem Q] we have 

Proposition 1. The combinatorial capacity C of (W, £} is 
given by the unique positive real solution of 

(|£|-l)G w (s) = l. (18) 

This proposition coincides with J5] Theorem 2]. As ob- 
served in Q, for £ = 2, the proposition reduces to iflOl 
Proposition 1.1]. Furthermore, if £ = {0,1} and W = 
{1,2,..., k}, then the proposition provides the combinatorial 
capacity of the (j = k, k) run-length constraint and coincides 
with the formulas derived in ]4|, ifTTI . 

VI. Probabilistic Capacity of (W, £) 

To calculate the probabilistic capacity of (W, £), it remains 
to define an input process for (W, £) that has an entropy rate 
equal to the combinatorial capacity C. 

A. Preliminary Considerations 

Not every process Y = {Kjg^ with Y t G (W, £) is a 
valid input process. The first reason is that a concatenation of 
elements from (W, £} is not necessarily an element of (W, £). 
Let, for instance, (W, £) be given by 

W := {1, 2, tt}, £ := {red, green}. (19) 

Define the run r by w(r) = n and l(r) = red. Run r is 
an element of (W, £). The string cat(r, r) is of color red, 
therefore, cat(r, r) is just one run. But w[c&t(r,r)] — 2ir ^ 
W, so the concatenation of r with r is not in (W, £} and Y 
violates condition 1) of Definition [5] 

A second reason why Y may not be an input process 
is ambiguity: define r\ by w{r\) = 1 and l(r\) = red 
and define by w(j2) = 2 and l{r2) = red. Obviously, 
T\,Tz S (W,C). But cat(ri,ri) = f2, so r2 can be a 
realization either of (Yi,!^) or just of Y\. This violates 
condition 2) of Definition |5] 

We leave the example and let from now on (W, C) again 
be an arbitrary general run-length set with \C\ > 2 and |W| 



possibly countably-infinite. We choose the following approach 
to construct an input process that guarantees both condition 
1) and condition 2) of Definition [5] we define a subset y 
of (W, C) such that if the Yj, take values in y, then both 
conditions from Definition [5] are automatically fulfilled. 

B. The Support of a Class of Input Processes of (W, C) 

We choose an arbitrary but fixed element Iq g £ and 
consider the subset y of (W, £) that consists of strings where 
the first run has label Iq and where all subsequent runs have 
a label different from Iq. Note that each string in y consists 
of at least two runs. The set y has two important properties: 

1) If x G y and y G y then cat(x, y) G (W, £), so 

OO 

Q{cat(a:i,...,z < )|(a:i,.-.,a:<) G y*} C (W,£). 

i=l 

2) Each string in (W, £) that starts with a run of label l 
and ends with a run of a label different from Iq can 
be unambiguously generated by concatenating elements 
from y. 

In other words, any random process {ii}^, Y{ G y fulfills 
condition 1) and condition 2) of Definition [5] and is therefore 
an input process of (VV, £). 

In the following derivations, we will also need the generat- 
ing function of y. It is given by 




from C\Iq but different from 
the label of the 2nd run 



(20) 

The derivation of Gy follows the lines of Section IIVI 

C. A Maxentropic Input Process of (W, £) 

For a random process that takes values in y as defined in the 
previous subsection, we now want to define a PMF in such a 
way that the entropy rate of the resulting input process is equal 
to the combinatorial capacity C and thereby maxentropic. To 
this end, we will need the following proposition: 

Proposition 2. Let Z = {Zi}°^ 1 denote a random process 
with each Zi taking values in the countably infinite set Z and 
let w: Z R>o be an associated positive weight function. 
Let R denote the unique positive real solution of 

G z (s) = 1 (21) 

and define pz as 

p z (z) := e- w ^ R , zeZ. (22) 

Then pz is a PMF and by letting the Zi be independent and 
identically distributed (IID) according to pz, 

R(Z) = R. (23) 

Proof: For a finite set Z, this proposition is equivalent 
to lfl2l Theorem 1]. By J4) Lemma 1] from our work, the 
generalization to countably infinite Z follows directly. ■ 



Let Y — {Yi}°^ 1 denote a random process where the Y 
take values in y, with y as defined in the previous subsection. 
We first show that Gy(C) = 1. Recall from (Qj) that 



|£|-1)G W (C) = 1. 



(24) 



We thus have 

oo 

G y (C) = G W (C) (|£| - 1) G W (C) V - 2) G W (C) 



fe=o 



=l-l-Gw(C) 



G w (C)^[l-G w (C)] fe 



fe=0 



Gw(C) 



1 - [1 - G W (C)] 
G W (C) 



= 1 



(25) 
(26) 

(27) 
(28) 



Gw(C 

where d25l l follows from d20l i. ( f26b follows from d24l i. and 
where we used the geometric series formula in (I271 l, By 
Proposition [2] 

-u)(y)C 



pv{y) 



y&y 



(29) 



is a PMF. We let the Yi be IID according to py. As a 
consequence, again by Proposition [2] 



H(Y) = C 



(30) 



Thus, the entropy rate of Y is equal to the combinatorial capac- 
ity C, and since C is according to Theorem |2] an upper bound, 
Y is maxentropic. We thus have the following proposition. 

Proposition 3. The probabilistic capacity of (W, C) is equal 
to its combinatorial capacity. 

D. Discussion of 

1 ) Definition of probabilistic capacity: Proposition [3] was 
proved in J5] with respect to the following definition of 
probabilistic capacity. Let Y = {Y}"^ denote an input 
process of (W, £). Let Y generate strings until time instant 
T, If the last run is not complete, discard the incomplete last 
run. Let vq denote the smallest element from W. The strings 
that Y can generate in this way are for each time instant T 
given by the set 

X T = {s e (W,C)\w(s) < T, w(s) + v > T}. (31) 

Define for each time instant T an auxiliary random variable 
Xt with the support Xt- Since the input process Y generates 
one of the strings from Xt until time instant T with probability 
1, the PMF of Y implies a PMF of X T . The authors of 
now define the entropy rate of Y by 



H(Y) := limsup 

T^foo 



H(Xz 
T 



(32) 



and define the probabilistic capacity of (W, C) by the max- 
imum of H over all input processes Y. With respect to this 
definition, it is easy to show Theorem [2] but it is difficult 



to show Proposition [3] This observation is reflected by the 
length of the proof of Theorem 4] given in (3). As a side- 
result of our work, both definitions of probabilistic capacity 
are equivalent, but their calculations differ. 

2) Maxentropic input process: In y], the authors solve 
the problem of violating the conditions from Definition by 
defining an input process with memory, specifically, a Markov 
chain with \C\ states. Basically, they consider PMFs of Y 
where 

Prob[/(Y +1 ) = t\l{Y t ) = t] = 0, Vt e C (33) 

i.e., two subsequent runs generated by the process have the 
same label with probability zero. It can be shown that the 
entropy rate as defined in Definition [4] of this process is 
equal to the combinatorial capacity C, i.e., this process is also 
maxentropic with respect to our definition of entropy rate. 

VII. Conclusions 

In this work, we used generating functions to show in a 
unified manner that the probabilistic capacity of a general run- 
length set is equal to its combinatorial capacity. This is an 
interesting result, since general run-length sets are in general 
not regular and can not be analyzed within the framework 
of finite state machines, which has been the usual approach 
for the analysis of constrained systems so far. Generating 
functions and their information-theoretic properties, which we 
recently established, appear to be a strong tool for the analysis 
of a broad class of constrained systems. We believe that known 
results can be re-established within this new framework in 
a unified manner and that new results can be obtained. The 
present work is a first step in this direction. 
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